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SUMMARY 


An  Artificial  Neural  Network  (ANN)  classifier,  based  on  fuzzy  Min-Max  [21],  Adaptive 
Resonant  Theory  (ART)  [2,5],  and  fuzzy  ART  [7,17]  is  described.  The  outputs  of  the 
classifier  are  fuzzy  hypercubes  representing  functional  categories  of  its  input  functions.  A 
hypothesis  and  test  paradigm  compares  input  data  and  existing  hypercube  categories,  and 
results  in  either  network  “resonance  or  dissonance,”  depending  on  the  test  outcome.  A 
hypothesis  is  formed  by  two  functions  which  measure  the  degree  of  match  between  input 
and  each  hypercube  category.  The  hypothesis  match  functions  are  Degree  of  Inclusion, 
and  Degree  of  Perfect  Match.  An  overall  hypothesis  is  chosen  based  on  the  best 
hypothesis,  i.e.,  the  hypothesis  which  has  the  greatest  match.  A  test  is  then  performed  to 
verify  the  hypothesis.  The  test  consists  of  both  a  vigilance,  and  overall  hypervolume 
limit  test.  The  vigilance  test  measures  the  top  down  match  between  the  hypothesized 
category  and  the  input.  The  overall  hypervolume  test  ensures  that  any  category 
adjustments  keep  the  total  category  hypercube  volume  within  bounds. 

Category  representation  is  extended  beyond  a  unit  hypercube  as  in  [21]  reflecting  the 
interpretation  as  a  degree  of  typicality,  rather  than  relative  [18]  to  allow  for  more  “noisy” 
feature  hypercubes.  The  network  has  7  layers;  input,  transform,  process,  hypothesize, 
test,  functional,  and  category.  The  network  is  described  in  a  hybrid  neuronal-functional 
approach. 

The  network  was  tested  with  standard  classification  test  data  for  recogniton  of  speakers  in 
an  open  set,  text-independent  environment.  The  standard  sets  were  Iris  and  Wisconsin 
Diagnostic  Breast  Cancer  (WDBC)  [26].  The  network  produced  88%  correct 
classification  for  WDBC  and  76%  for  Iris. 

A  speaker  recognition  system  [12,13,14]  using  a  fuzzy  hypercube  classifier  was  tested 
using  the  Switchboard  [22]  and  Greenflag  [23]  data  bases.  The  fuzzy  hypercube  ANN, 
characterizing  one  speaker  per  category,  produced  an  average  of  6.29  correct  and  0.29 
incorrect  categories  out  of  a  possible  8  total,  with  no  prior  training.  The  overall  average 
correct  classification  produced  by  the  network  using  a  fixed  set  of  signal  features  for  8, 
12,  and  16  speaker  groups  was  67%.  See  [15]  for  details. 
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1.  INTRODUCTION 


Classification  is  the  process  of  labeling  data  as  groups  which  are  related  hy  one  or  more 
common  concepts.  There  are  numerous  approaches  and  definitions  of  classification  in 
the  literature  based  on  statistical  methods.  See  [8]  for  an  introduction.  Fuzzy  methods  in 
classification  are  developed  in  [1],  along  with  categorization  of  classifier  designs 
according  to  model,  data,  output,  decision  regions,  algorithm,  and  architecture.  In  this 
report,  we  are  concerned  with  designs  which  are  based  on  the  general  hypothesize  and 
test  paradigm  with  data  driven  hypothesis  formation  [24].  Here  the  knowledge  is 
decomposed  hierarchically  into  “levels  of  analysis”  [24]. 

The  use  of  an  Artifical  Neural  Network  as  a  classifier,  particularly  for  unsupervised 
learning,  is  based  on  the  architectures  of  ART  [2,3,4],  fuzzy  ART  [6,17],  and  fuzzy  min- 
max  [21],  These  all  employ  the  general  hypothesis  and  test  paradigm,  but  are  more 
neuronal  “model-based”  methods.  In  each  the  general  processes  of  hypothesis  generation 
are  roughly  maintained  as  a  neural  structure,  and  a  testing  mechanism  is  described  as  a 
somewhat  complex  feedforward/feedback  functional  control  mechanism. 

A  brief  introduction  to  the  basic  and  fuzzy  ART  functions,  architectures,  standard 
terminology,  and  equations  will  be  presented  next  as  background.  Familiarity  with  basic 
fuzzy  sets,  fuzzy  set  theory  and  clustering  is  assumed.  See  Chapter  2  in  [1]  for 
introductory  papers  in  fuzzy  set  theory  and  clustering,  and  Dubois  and  Prade  [9]  for 
further  details  on  general  fuzzy  set  theory. 

1.1  ART2 

A  typical  ART2  [14]  network  is  composed  of  two  layers  (or  fields)  of  neurons  which 
form  the  Attentional  Subsystem.  The  first  field  is  the  Feature  Representation  Field,  or 
Fi,  which  contains  processing  elements  (PE)  that  form  three  intra-PE  sublayers  which  are 
responsible  for  processing  one  element  in  the  input  pattern.  The  main  function  of  the 
feature  representation  field  is  to  enhance  the  current  input  pattern's  salient  features  while 
suppressing  noise. 

The  second  layer  in  the  attentional  subsystem  is  called  the  Category  Representation 
Field,  or  F2,  which  represents  one  category  (or  class)  that  has  been  learned  by  the 
network.  The  connections  from  a  particular  F2  neuron  store  a  category  pattern.  ART2 
utilizes  an  unsupervised  learning  technique  which  attempts  to  discover  the  distributions 
and  centroids  of  the  categories  for  the  patterns  it  is  presented.  ART2  can  utilize  a 
"winner-take-all"  classification  strategy,  such  as  MAXNET  [19].  Having  selected  the 
winner,  an  Orienting  Subsystem  is  activated  to  determine  whether  the  proposed  winning 
neuron's  Long  Term  Memory  (LTM)  traces  sufficiently  resemble  the  Short  Term 
Memory  (STM)  pattern  to  be  considered  a  match.  A  matching  threshold  called  the 
Vigilance  Parameter  determines  how  similar  the  input  pattern  must  be  to  the  exemplar  to 
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be  considered  a  match.  If  the  degree  of  match  computed  by  the  orienting  subsystem 
exceeds  the  vigilance  parameter,  a  state  of  resonance  is  attained  and  the  STM  pattern  at 
F]  is  merged  onto  the  winning  neuron's  LTM  traces.  Otherwise,  the  orienting  subsystem 
sends  a  reset  signal  to  the  winning  neuron,  and  inhibits  it  from  competing  again  for  the 
current  input  pattern.  This  search  process  is  repeated  until  either  an  F2  neuron  passes  the 
vigilance  test  or  all  established  F2  neurons  have  failed  the  test.  In  the  latter  case,  a  new 
category  is  established  in  the  next  available  F2  neuron. 


Learning  is  competitive  with  each  F2  neuron  attempting  to  include  the  current  input 
pattern  in  its  category  code.  The  actual  learning  process  involves  modification  of  the 
bottom-up  and  top-down  LTM  traces  that  join  the  winning  F2  neuron  to  the  feature 
representation  field.  Learning  either  refines  the  code  of  a  previously  established  class, 
based  on  any  new  information  that  is  contained  in  the  input  pattern,  or  initiates  code 
learning  in  a  previously  uncommitted  F2  neuron  [2], 

In  either  case,  learning  only  occurs  when  the  system  is  in  a  resonant  state.  This 
property  ensures  that  an  input  pattern  does  not  obliterate  information  that  has  been 
previously  stored  in  an  established  class. 

1.2  Fuzzy  ART 

The  basic  operation  of  “adaptive  resonance”  in  the  standard  ART  is  simplified  in  the 
fuzzy  ART.  The  basic  equations  which  govern  the  fuzzy  ART  are  based  on  the  equations 
from  the  standard  ART  architecture,  where  the  intersection  operator  is  replaced  by  its 
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fuzzy  counterpart,  the  minimum  operator.  An  introduction  of  the  mathematics  governing 
the  fuzzy  ART  is  given  here,  based  on  [4,5,6,7,25] 

The  fuzzy  ART  system  consists  of  three  layers:  the  input  layer  (FO),  processing  layer 
(FI),  and  output  category  (F2)  layer.  Associated  to  connections  between  layers  FI  and 
F2  are  a  set  of  weights  directed  from  FI  to  F2.  A  fundamental  difference  between  the 
Fuzzy  ART  and  prior  continuous  versions  is  the  simplification  of  the  “resonance  criteria” 
by  use  of  only  bottom-up  weights  in  the  matching  process.  The  matching  process 
consists  of  two  vector  matching  operations: 

•  Degree  which  input  A  matches  output  category  c 

•  Degree  which  category  c  matches  input  A 

The  norm  of  a  vector  A,  which  gives  an  indication  of  its  “size,”  is  defined  as 

14= EM  •  <» 

The  following  three  sections  describe  the  fuzzy  mathematical  functions,  operations  and 
data  structures  associated  within  each  of  these  layers. 


1)  Input  Laver 

Given  an  input  vector  A,  A  =  (a, }  or  optionally,  with  the  complement 

A  =  {aj,aCj  },  j  =  l,2,...,Nin  (2) 


The  addition  of  the  complement  of  the  input  vector  has  the  advantage  that  A  is  now  self- 
normalized,  using  the  definition  of  norm  in  Eq.  1 : 


,  "in  "in  "in 


(3) 


7=1 


7=1 


7=1 


2)  Output  Laver 

The  output  layer  F2  consists  of  a  set  C  of  Nmax  active  categories, 

C=  {Cp.-.jC^  } 

Each  category  vector  Cj  eC  has  an  associated LTM  weight  set 

WJ  =  {>V>W2,7>"-W2tfi„,7} 

3)  Processing  Laver 

A  category  Choice  Function  T j  measures  the  degree  which  input  A  is  a  match  to  a  Cj 
and  its  associated  Wj  : 


AnWj 

I 

1 

<*+ |K 

a  + 

Wj 

where  a  >  0  is  a  choice  parameter. 

T  is  the  best  category  choice,  and  is  calculated  as  the  union  of  all  Tj . 
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(5) 


T  =U^  =  MAX(Tj  ) 

J  j 

There  are  two  cases  which  can  occur  once  a  category  choice  is  attempted: 

Case  1.  Equation  5  produces  a  choice  J.  A  test  is  performed  on  the  preliminary  choice  J 
to  test  if  it  meets  a  threshold  criteria  called  the  vigilance  test,  where  the  degree  to  which 
the  preliminary  category  matches  the  input  A  is  compared  against  a  threshold  p 

AnWj  |  \MIN{A,Wj)\ 

"  PI 

If  the  vigilance  criteria  of  Eq.  6  is  not  met,  the  preliminary  choice  [J]  is  said  to  be  “reset,” 
and  another  category  choice  according  to  Eqs.  4  and  5  is  made  from  the  set  of  active 
categories  in  C. 

If  the  vigilance  criteria  is  met,  then  the  system  is  said  to  be  in  a  state  of  resonance,  and 
the  input  A  is  incorporated  into  category  J  by  the  following: 

+  (7) 

Fast  learning  is  said  to  occur  when  /?  =  1 . 

Case  2.  Equation  5  produces  no  choice.  If  no  category  choice  can  be  made,  a  new 
category  CN+]  is  created  in  C  with 


w 


new 

N+\ 


=  W 


old 

W+l 


=  A. 


Initialization:  N=0 


(8) 


A  simplified  fuzzy  ART  architecture  is  described  by  Kasuba  [17].  In  the  remainder  of  this 
report,  we  will  describe  a  classifier  based  on  ART2,  fuzzy  ART,  simplified  frizzy  ART, 
and  fuzzy  hypercube  ART.  Section  2  will  introduce  and  define  the  fuzzy  hypercube 
classifier.  Section  3  will  compare  simplified  fuzzy  ART  and  fuzzy  hypercube  ANN  using 
standard  classification  sets.  Section  4  will  give  an  application  of  the  fuzzy  hypercube 
classifier  to  Speaker  Recognition,  and  Section  5  will  briefly  summarize  the  results. 
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2.  FUZZY  HYPERCUBE  ARTIFICIAL  NEURAL  NETWORK 


Modifications  to  ART  and  Fuzzy  ART  were  done  to: 

1)  Improve  performance  since  they  generally  suffered  from  poor  noise  tolerance 

2)  Provide  workable  functional  basis  for  hypothesis/test  of  neural  systems 

The  Fuzzy  Hypercube  ANN  (FHANN)  is  a  conceptual-neural  model  based  on  the 
following  general  features: 

1)  Hypothesize  and  Test  paradigm, 

2)  Hypercube  category  (concept)  representation 

3)  Category  Overlap 

4)  Fuzzy  Information  representation  and  processing 

5)  Information  Fusion  (optional) 

6)  Category  Match  Test 

7)  Hypervolume  category  limit 

An  FHANN,  which  implements  the  above  general  features,  will  be  described.  First  an 
overview  will  be  given  of  the  network  layer  structure,  followed  by  a  detailed  description 
of  each  layer.  Also,  global  category  merge  and  ANN  initialization  are  described. 


2.1  Fuzzy  Hypercube  ANN  Structure 

The  FHANN  has  seven  layers  of  processing.  Figure  2  shows  the  functional  structure, 
interconnection,  and  data  flows  of  the  ANN  structure.  Notice  the  network  bi¬ 
directionality  in  its  processing  and  information  flows.  Specific  category  information  is 
fed  back  to  the  Fusion  and  Transform  layers  for  state-dependent  information  adjustment, 
and  to  the  Functional  layer  to  performing  category  adjustment  and  learning. 

The  function  of  each  of  the  seven  layers  in  the  FHANN  is: 

Input :  Input  fuzzification,  scaling,  optional  functional  expansion 

Transform :  Evaluation  of  category  choice  functions  over  active  categories 

Fusion:  Fusion  to  final  ratings  of  category  choice  evaluation  [Optional  Config] 

Hypothesize-.  Choice  of  single  hypothesis  category 
Test-.  Pass/fail  match  test  of  hypothesis  category 

Functional:  Category  creation,  hypervolume  adjustment,  or  category  learning 

Category:  Representation  of  clusters  as  fuzzy  hypercube 

The  FHANN  has  several  distinct  differences  from  the  basic,  fuzzy,  and  simplified  fuzzy 
ART.  It  retains  the  basic  data  structures  using  A  and  x  vectors.  The  concept  of  bottom 
up  and  top  down  match  and  the  learning  rule  are  different. 

Each  of  the  layers  are  described  in  the  following  sections.  A  detailed  view  of  the 
network  is  shown  in  Figure  3,  where  each  of  the  blocks  from  Figure  2  is  expanded  to 
show  more  details. 
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2.1.1  Input  Layer 

The  Input  layer  has  three  inputs  and  3  *  Nln  outputs.  The  inputs  are  vector  A,  scaling 
values  for  A,  and  enable/disable.  The  vector  A  and  associated  scaling  generate  fuzzy  set 
representations  of  the  input  as  “Feature  Vector”.  The  enable/disable  inputs  control  the 
Input  layer  and  overall  network  resonance/dissonance. 

2.1. 1.1  Fuzzy  Set  Generation 

The  input  vector  and  its  optional  complement  are  described  by  Eqs.  2  and  3. 

A  scaling  function  is  defined  based  on  the  range  of  inputs  between  an  expected  minimum 
and  maximum  for  each  element  of  the  input  set  A.  The  scaling  function  maps  input 
values  to  the  closed  interval  [0,1]. 

A  discrete  fuzzy  set  “Feature  Vector”  (FV)  results  from  the  set  of  mapped  input  values  of 
A.  This  fuzzy  set  characterization  is  represented  by  a  crisp  set  of  individual  membership 
functions  s(a),  one  for  each  input  feature  value  a.  Note  that  this  standard  representation 
of  a  fuzzy  set  uses  the  plus  sign  to  indicate  the  union  of  the  elements. 

ITT/  _  v(“l )  ,  s(a2 )  ,  ,  ) 
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Figure  3.  FHANN  Functional  Diagram 
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The  function  s  translates  input  features  into  membership  values. 


s{aj)  = 


where 


0, 

aj  S  FMin  J 

*  z  j  ’ 

F Mmj  a  j  —  F Max 

1, 

ai  >  FMaxJ 

F 

1  MinJ 

I  ^ Maxj  ^ MinJ  I 


overall  j  =\,2,-,Nin  (9) 


Individual  values  of  FMittFMax  are  initialized  as  either 

•  Constants 

•  Learned  Variables 


A  simple  weighted  learning  procedure  can  be  implemented  to  dynamically  adjust  the 
values  of  FMin,  FMax  .  Although ,  in  this  report,  only  constant  values  are  used. 

2. 1.1. 2  Layer/Network  Resonance/Dissonance  Control 

The  enable/disable  input  controls  iteration/search  cycling  in  the  network.  The  network 
either  continues  to  cycle  with  current  input  A  when  a  suitable  category  is  not  found  by  the 
Hypothesize/Test  layers,  or  stops  cycling  and  begins  processing  of  the  current  input, 
while  enabling  the  acceptance  of  the  next  input. 

2.1.2  Category  Layer 

The  Category  layer  produces  the  output  of  the  network  and  consists  of  a  set  of  neurons 
with  associated  states  and  LTM  weight  values  which  descibe  them.  The  LTM  weights 
are  described  using  a  min-max  feature  hypercube  representation  of  the  associated  J- 
categories  defined  by  Simpson  [21].  Strictly  speaking,  the  categories  are  not  hypercubes 
but  “hyperboxes,”  since  they  do  not  have  equal  dimensions. 

The  category  layer  C  is  a  set  defined  by: 

c  =  {^,vy,rJ,^},  j  =  1,2,...,  vmax 

'  (  (10) 
with  BJ  =  {Vj,Wi\,  j  =  \,2,...,N jn  ,  e [0,1] 

where  Nmax  the  total  number  of  J-categories,  Njn  is  the  j -dimensionality  of  the  hypercube 
representation  according  to  Eq.  2. 

The  category  layer  consists  of  BJ  as  the  hypercube  representation  of  category  J.  BJ 

contains  a  jth  dimensional  description  of  Vj  the  minimum  point,  and  Wj  the  maximum 
point.  Additionally,  category  J  consists  of: 

•  NJ  is  a  count  of  category  adjustments 


8 


•  TJ  is  an  overall  confidence 

•  SJ  is  the  state  of  category  J 

2.1.3  Transform  Layer 

FV  values  from  the  Input  layer  are  passed  to  the  Transform  layer,  where  they  are 
processed  by  a  match  function  and  generate  a  degree  of  match  between  the  input  A  and 
each  member  of  the  output  category  set  C.  A  membership  of  the  input  in  each  of  the 
output  category  classes  is  produced. 

A  wide  range  of  choices  is  available  for  the  matching  functions  such  as  those  in  ART2  or 
fuzzy  ART  (Eq.  4).  Several  authors  examined  various  fuzzy  match  functions.  Eq.  4  has 
been  expanded  by  Carpenter  and  Gjaja  [7]  to  Choice-by-difference,  providing  a  more 
conservative  learning. 

Tj  =  (\Wj\-\AAWJ\)  +  s(\l^WJ\-\Wj\) 
where  s  is  analogous  to  a  of  Eq.  4. 

Simpson  [21]  developed  a  membership  function  which  measures  the  average  amount  of 
max  point  and  min  point  violations.  He  defines  a  function  b  which  approaches  1  as  the 
point  approaches  a  hypercube, 

b„  (■ Ah,Vj,Wj)=  -^-£[1-  -wM>r)-  f{va  ~ah,,r)] 

™  in  i=  1 


where  f()  is  the  ramp  function, 

fl  if  xy>  1 


f(x>r)  =  I 


xy 

n 


if  0  <  xy  <  1 


The  variable  y  regulates  the  speed  of  membership  decrease  when  an  input  is  separated 
from  a  hyperbox  core  [21]. 

The  idea  of  a  smooth  membership  function  which  approaches  one,  as  a  point  is  within  a 
target  hypercube  and  steadily  decreasing  as  it  leaves  the  hypercube,  has  conceptual 
support.  However,  the  function  bb  above  does  not  provide  a  clear  indication  of  goodness 
once  inside  the  hypercube. 

2.1.3.1  Matching  Functions. 

A  general  and  exact  hypercube  match  is  formulated  as  the  linear  combination  of  two 
functions  [11].  The  functions  measure  fuzzy  hypercube  “Degree  of  Inclusion”  (DOI)  and 
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“Degree  of  Perfect  Match”  (DPM).  They  are  combined  to  give  a  measure  of  the  level  at 
which  the  scaled  input  matches  to  each  feature  category  hypercube. 


2.1.3.2  Degree  of  Inclusion 
u(x,y) 


2 

Y 


Figure  4.  Trapezoidal  Degree  of  Inclusion 


DOI  measures  the  level  at  which  each 
input  Aj  matches  to  the  external  dimensions 
of,  or  is  similar  to,  the  j-th  dimension  of  a  J- 
category  hypercube. 

This  method  is  similiar  in  function  to 
Simpson’s  [21].  A  trapezoidal  function  is 
used  as  a  model  for  a  match  which  gives 
full  membership  when  an  element  of  Aj  is 
included  in  a  category,  and  less  than  full 
membership  outside,  depending  on  the 
distance  to  the  hypercube.  Figure  4  shows 
the  trapezoidal  membership  function. 


The  set  H,  consisting  of  upper  and  lower  fuzzification  values  is  defined  for  each 
hypercube  BJ ,  as 


HJ  ={BJ,LJ,MJ),  J  =  l,2,...,NMax 

LJ  =  {Lj},  MJ  =  {Mj},  j  =  \,2,...,Nhl  (11) 

where  the  sets  LJ ,  MJ  represent  the  fuzzification  sets  for  the  upper  and  lower  limits. 

The  overall  membership  for  DOI,  // 1)01  (a)  is  defined  for  each  j -dimension  of  the 
subsets  of  HJ  for  hypercube  J  (Eqs.  10,1 1)  as  a  trapezoidal  function  //  D°I  (A) . 

IF  Wj  >aJ>Vj  /urf‘(A)  =  \ 

IF  Mj  >aj>Wj  =  1  - (dj  - Wj)y  (12)' 

IF  Vj  >  a  j  >  Lj  =1  -(Vj-aj)/ 

IF  Lj  >  dj  OR  Mj  <dj  =0 

As  a  practical  matter  we  set  | lj  —  vy  |  =  |wy  -  m^  =  j  to  evenly  fuzzify  the  hypercube.  The 
value  —  is  chosen  as  a  constant  range  of  “fuzzy”  border  around  V  and  W  for  each  value 

r 

of  A. 

The  oveall  membership  function  for  the  DOI,  n  DO‘  ( A  ) ,  is  defined  as  the  following: 
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(13) 


A 


DOI 


DOI 

j 


Note  that  0  <  £  //  f0/  <  Nin 


2.1.3.3  Degree  of  Perfect  Match  (DPM) 

The  DPM  measures  how  close  an  input  is  to  the  mean  of  each  J-category.  The  measure 
of  the  distance  from  the  mean  of  each  dimension  of  Hj  is  defined  as  DPM.  DPM  is  a 
resemblance  relation  between  the  input  A  and  an  individual  J-category.  First,  the  mean 


MJ  of  a  category  J  is  defined. 


MJ  =  {rrij}  = 


7  =  1,2, 


(14a) 


The  dissimilarity  is  defined  as  the  difference  between  the  value  A  and  the  mean  of  the 
category  J,  MJ , 


Dissimilarity  =\A-  MJ\ 


and  for  each  j -dimension  of  A  and  MJ , 


Dissimilarity  j  =  a  t  -  mj 


The  similarity  is  taken  as  the  fuzzy  complement  of  the  dissimilarity, 

Similarity  =  Dissimilarity j 
and,  for  each  j -dimension  element,  we  have 

Simj  =  1  - 1  Oj  ~  m}  |  (14b) 


Using  the  mean  and  similarity  from  Eqs.  14a  and  14b,  the  individual  j -dimension 
membership  functions  for  DPM,  //  DjPM  (a  f ) are  defined  as  follows 


i-h -VA 

1  -\aj  -m\ 


if  Wj  =  Vj 

if  Wj  *  Vj 


(14c) 


The  overall  membership  function  // DPM  (A)  for  a  single  J-category  is  the  mean  of  the 
sum  of  the  individual  values, 


N, 


(15) 


in  J 


Again,  note  that  0  <  J>f"'(a,)  <  W.,  and  Mj  (A)  e  [0,1] 
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2.1.4  Fusion  Layer 

The  Fusion  layer  integrates  the  best  knowledge  in  the  network  concerning  the  matching 
between  the  current  input  A  and  each  of  the  current  categories.  The  knowledge  used  in 
the  matching  process  is  the  DOI  and  DPM  from  the  Transform  layer,  as  well  as  certain 
feedback  category  information  that  comprise  the  input  to  the  Fusion  layer.  A  fusion 
function  rj  is  defined  as  the  linear  combination  of  the  DPM  and  DOI  that  are 
dynamically  weighted.  The  dynamic  weighting  is  done  to  compensate  for  low  DOI  at  the 
start  of  a  hypercube  matching  cycle,  and  is  based  on  the  number  of  inputs  representing  a 
resultant  category. 


RJ(A)  =  k;flr(A)  +  k!fl?'(A) 

(16a) 

where 

k{  =  min (K  *  NC(  J),\) ,  kf  +  k{  =  1 

(16b) 

VI 

*-> 

0  <  K  <  1 ,  K  constant 

The  functions  k(  ,k{  are  dynamic  weight  functions  which  give  the  relative  importance  of 
DPM  and  DOI  to  the  value  RJ .  Note  that  DPM  and  DOI  are  weighted  complementary, 
to  allow  for  RJ  <  Nin . 

The  node  constant  NC  is  a  dynamic  weight  function  dependent  on  NJ  in  C,  which  is  the 

number  j  of  input  vectors  which  are  assimilated  through  learning  into  a  J-category 
hypercube. 


NC(J)  = 


0.65, 

NJ  =  1 

0.85, 

NJ  =2 

0.95, 

NJ  =3 

1.00, 

NJ  >3 

(17) 


2.1.5  Hypothesis  Layer 

Inputs  from  the  Fusion  layer  form  a  hypothesis  space  from  which  a  single  winnning 
category  hypothesis  is  either  chosen  or  no  winner  is  chosen  meaning  no  current  category 
fits  the  current  input  and  a  new  category  is  then  created.  Hypotheses  are  formed  and  a 
Winner/No  Winner  is  chosen  by  first  finding  the  maximum  over  all  the  active  J-category 
fusion  values,  RJ .  An  active  category  is  one  which  has  not  yet  been  processed  using  the 
current  input  as  a  hypothesis;  if  it  has  already  been  processed  it  is  marked  inactive. 

Winner:  Cw  =  max{RJ}if  RJ  is  active  and  RJ  >  0,  J  =  l,...,Nmax 
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No  Winner,  if  RJ  is  inactive , or  RJ  <  0,  overall  J 

The  resultant  hypothesis  “  Winner  ”  is  passed  with  the  winning  category  node  to  the  Test 
layer,  while  a  uNo  Winner”  hypothesis  is  passed  back  to  the  Input  layer  to  halt  resonation 
of  the  network,  as  well  as  to  notify  the  Functional  layer  to  create  a  new  category  node  for 
the  current  input  A. 

2.1.6  Test  Layer 

The  Test  layer  performs  a  match  test  on  the  curent  active  winning  category  Cw  as  an 
input  hypothesis.  The  value  of  r  3  associated  with  cw  measures  the  degree  of  match 
between  the  category  hypercube  b  and  the  input  vector  A.  A  modified  form  of  the 
vigilance  test  (Eq.  6)  is  performed  as  shown  in  the  following  equations  to  measure  if  the 
current  hypothesis  passes  the  vigilance  threshold. 

RJ  >  padJ  (18) 

where 

padi=pNC(J)  (19) 

and  NC(J)  is  described  by  Eq.  17. 

Additionally,  any  current  category  which  fails  the  vigilance  test  is  inhibited  from 
competing  with  the  current  input  by  making  it  inactive. 

2.1.7  Functional  Layer 

The  Functional  layer  performs  a  series  of  services  on  the  Category  layer  including 
Hypervolume  Measure,  Hypervolume  Test,  Hypervolume  Adjust,  Hypercube  Learning, 
and  Hypercube  Creation. 

2.1.7.1  Hypervolume  Measure 

There  are  two  methods  described  to  calculate  hypervolume:  by  a  product  and  by  a  sum. 
The  product  hypervolume,  Phv,  which  corresponds  to  the  “geometric”  interpretation  of 

volume,  is  found  by  the  product  of  the  “length”  in  each  dimension  of  Bj . 

Phv  =\f[(Wt  -V,  )\  (20) 

1—1 

where  0  <  Phv  <  1 

The  product  measure,  however,  does  not  allow  for  much  expansion  in  the  measure  of  a 
hypercube  with  variable  length  concepts  or  inclusion  of  noise,  etc.  Another  measure  of 
the  hypervolume  which  increases  the  effect  of  each  dimension  is  the  unweighted  sum  of 
each  of  the  hypercube  dimensions,  Shv, 

Sin 

Shv  =  Y,\Wi -Vt\  (21) 

7  =  1 

where  0  <  Shv  <  Nm 
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2. 1.7.2  Hypervolume  Test 

Hypervolume  limit  testing  and  adjustment  are  necessary  for  the  stability  of  the  network. 
In  FHANN,  for  each  J-category,  the  hypervolume  measure  is  constrained  to  be  less  than  a 
maximum  limit,  ©max .  The  hypervolume  ©  of  a  hypercube  is  bounded  to  keep  it  from 
expanding  to  infinite  volume. 

0  <  0  <  ©max  for  each  J  J  =  l,...,Amax  (22) 

where  the  operation  which  determines  ©  is  selected  from  the  set  of  hypervolume 
measure  operations  defined  by  Eq.  19  or  Eq.  20. 

©  =  {Phv,Shv} 

2.1.7.3  Hypervolume  Adjust 

If  the  limit  0max  is  exceeded,  the  category  hypervolume  is  adjusted.  The  excessive 
volume  A®  is  found  from  the  current  hypervolume,  0 ,  by  the  following: 


A©J(0-®' 


(23) 


)/  N.  ©  >  © 

t  /  w  ^  max 

v  0  Otherwise 

where  Nm  is  the  current  input  dimensionality  as  in  Eq.  1 1.  The  hypervolume  of  category 
J  is  adjusted  whenever  A®  >  0  by 


Wfw  =  ma x{{W°,d  -  A©  /  2),0} 

Vjnew  =  min  {(Vj°,d  +  A©  /  2),1}  (24) 

This  operation  brings  the  hypervolume  measure  of  category  J  within  the  value  of  ©max 
as  required  by  Eq.  22. 

2. 1.7.4  Hypercube  Learning 

The  inclusion  of  input  A  into  the  winning  category  hypercube  Bj  is  done  through  a 
learning  algorithm  which  adjusts  the  category  J  hypercube.  In  general,  each  feature  of  A, 

a, ,  selectively  adjusts  its  respective  limits  in  W,  and  Vj  through  the  following  hypercube 

learning  algorithm.  A  learning  adjustment  factor  r  is  used  to  set  the  algorithm’s  rate  of 
learning. 


Hypercube  Learning  Algorithm.  Given  an  input  vector  A,  a  hypercube  B„  and  a  learning 
adjustment  factor  r,  learning  is  performed  for  each  dimension  of  A. 


1 .  Determine  if  input  is  inside  or  outside  of  interval  Bj 

2.  DO:  Respective  Case  1  or  Case  2  and  subcases  below. 


Case  1 :  Input  is  outside  of  interval  Bj 

Case  1.1  Input  above  W,:  Increment  W  upwards 


14 


IF  a,  >  Wf  THEN  Wf  =  Wf  +r(a,  -  Wf ),  Vmw  =  VM 
Case  1.2  Input  below  Vj:  Decrement  V  downwards. 

IF  a,  <  Vf  THEN  V™  =  Vf  +  r{a,  -  Vf ) ,  Wnew  =  W°,d 


Case  2:  Input  is  inside  of  interval  B j 

Case  2. 1  Input  is  closer  to  V;  move  W  closer  in  to  V. 

IF  (a,  >  Vf  AND  a,  <  Wf  AND  Wf  -  a,.  >  a,.  -  Vf) 

THEN  W'jr  =  Wf  -  r(Wf  -  au ) ,  V'f  =  Vf 

Case  2.2  Input  is  closer  to  W;  move  V  closer  to  W. 

IF  (a,  >  Vf  AND  a,  <  Wf  AND  a,  -  Vf  >  Wf  -  a , ) 

THEN  Vf  =  Vf  +  r(a,  -  Vf ) ,  Wf  =  Wf 


2.1.7.5  Hypercube  Creation. 

The  creation  of  a  hypercube  requires  that  the  overall  hypervolume  limit  is  adjusted 
through  the  hypercube  dimension,  hdmax,  which  is  defined  as  the  maximum  hypercube 
dimension,  assuming  equal  size  in  each  dimension: 


0 

=  raax 


N, 


Upon  Creation: 

1.  Initial  settings  are  equal. 


Wf  =  Vf  =  at 

2.  Number  of  active  J-categories  is  incremented  by  one. 


(25) 


Nmax=Nm  „  +  ! 


2.2  Auxiliary  Network  Functions 

The  FHANN  has  optional  and  required  auxiliary  functions.  The  required  functions 
include  FHANN  Initialization  of  system,  and  FHANN  parameter  initializations.  Optional 
function  is  the  Global  Category  merge  which  was  tested,  but  results  are  not  reported  here. 


2.2.1  Category  Merge 

A  global  merge  is  defined  as  a  combination  of  hypercube  cluster  classes  produced  by  the 
FHANN  which  are  very  “close”  to  one  another.  This  operation  is  performed  outside  of 
the  neural  network  processing  and  does  not  affect  any  of  the  internal  operations  of  the 
network.  It  does  utilize  detail  parameters  generated  by  the  network. 


15 


This  process  occurs  over  time  between  FHANN  cycles  and  can  be  considered  a  long  term 
averaging  process.  There  are  two  measures  which  are  used  to  indicate  whether  a  global 
merge  is  to  take  place: 

a)  Volume  difference  between  hypercube  categories  and 

b)  Magnitude  of  rating  R  from  Eq.  1 6  between  two  categories. 

The  volume  parameter  is  defined  as  follows: 

Nm  x  A  <  volume  (26) 

The  value  Nm  is  the  number  of  input  nodes  and  A  the  hypervolume  per  node. 

A  category  merge  function  is  defined.  First,  merge  parameters  are  obtained  over  all 
possible  different  pairs  of  the  current  categories  defined.  Next,  the  merge  criteria  are 
applied  and  partition  current  categories  into  a  final  set  which  is  compacted.  [Note  that 
during  testing,  the  compacting  occurred  very  rarely].  The  criteria  are  expressed  in  terms 
of  acceptance/rejection  regions  in  the  volume  difference/rating  mapping. 

0.0  <  Avo/(cl,c2)  <  1.10 and  R{c\,c2 )  >  1.00  OR 

1.1  <  Avo/(cl,c2)  <  l.50and  R(cl,c2)  >  1.00  OR  (27) 

1.5  <  Avo/(cl.c2)  <  1.75am/  R{c\,c2 )  >  1.40 
These  were  experimentally  derived  and  were  only  used  to  evaluate  the  concept  of  global 
clustering  criteria  within  the  context  of  the  hypercube  structure. 

2.2.2  FHANN  Initialization 

The  initialization  is  performed  on  the  FHANN  as  follows: 

Step  1 .  Enable  all  categories,  set  count,  and  confidence  is  “none”. 

NJ  =0,  TJ  =  none,  SJ  =  enabled 
Step  2.  Set  available  category  count  to  zero. 

N avail  =  0 

2.2.3  FHANN  Parameters 

The  FHANN  contains  several  variables  and  parameters  which  are  under  the  control  of  the 
designer  of  the  system.  The  parameters  are  to  be  set  before  NN  execution,  and  are  as 
follows,  with  references  to  defining  equations: 

PI:  Size  of  input  layer,  Njn  (Eq.  2) 

P2:  Maximum  number  of  output  categories,  A/max(Eq.  10) 

P3 :  Complement  of  input  A  in  input  layer  (Eqs.  2,  3) 

P4:  Maximum,  minimum  input  scaling  values  in  A,  F^F^  (Eq.  9) 
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P5:  Hypercube  fuzzification,  A ,  for  fuzzy  border  around  each  a}  (Eq.  12) 

r 

P6:  Fusion  function  weight,  K,  the  fraction  of  DPM  (Eq.  16b) 

P7:  Base  value  of  vigilance,  p  (Eq.  19). 

P8:  Hypervolume  maximum,  hvmm  (Eq.  22) 

P9:  Learning  adjustment  factor,  r  (in  Hypercube  Learning  Algorithm) 

PI 0:  Hypercube  Measure  Type  (Eq.  19  or  Eq.  20) 

One  optional  parameter,  if  known,  is: 

01:  Known  Number  of  input  classes,  Nclms 


3.  NETWORK  COMPARATIVE  PERFORMANCE  ANALYSIS 

The  performances  of  fuzzy  ART  and  the  FHANN  are  compared  using  two  standard 
classification  data  sets:  Iris  Flower  and  Wisconsin  Diagnostic  Breast  Cancer.  A 
simulation  of  the  FHANN’s  was  made  using  MATLAB.  This  section  provides  details  on 
the  structures  for  the  simulator  and  results  of  the  comparative  testing  performed. 

3.1  Test  Measurement  Definitions 

Test  data  items  which  are  measured  to  quantify  the  performance  of  the  FHANN’s  are 
defined  in  this  section.  Note  that  for  all  data  tested,  the  correct  responses  are  known  but 
are  used  only  in  the  calculation  of  the  ANN  performance. 

The  definitions  for  CC,  TC,  and  IC  are  test  measurements  made  on  the  FHANN’s  of  the 
number  of  categories  created  by  the  networks. 

CC=“Correct  Category”:  Count  of  categories  containing  data  designated  as 
correctly  classified. 

TC- ‘All  Created  Categories”:  Count  of  all  categories  created 
IC=TC-CC- Incorrect  Categories”:  Count  of  categories  containing  data 
designated  as  incorrectly  classified 

The  definitions  for  the  following  are  test  measurements  made  on  the  FHANN’s  of  the 
number  of  input  data  elements  placed  in  the  various  classes  defined  as  CC,  TC,  or  IC. 
The  prefix  function  “D”  indicates  the  measure  of  data  in  a  specific  category. 

D=“Data”:  Function  indicating  individual  data  sets  in  a  specific  class/category. 
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D(CC)=Data  in  Correct  Category 
D(TC)=Data  in  All  Created  Categories 
D(IC)=Data  in  Incorrect  Categories 

Correct  Classifications  Vs  Incorrect  Classifications  in  “Correct  Categories  ” 

One  measure  of  the  performance  of  the  FHANN  being  used  is  the  fraction  of  how  many 
of  the  data  in  the  created  categories  are  correct  or  incorrect,  with  respect  to  the  sum  of  the 
correct  and  incorrect  data  input  to  the  system.  These  are  defined  by  Eqs.  28  and  29. 

2D(CO 

P(T  =  £{Z>(CO  +  Z>(/C)}  (28) 


Iw 

Pc  - _ 

“  Y.mcc)+D(ic)} 

Note  that  Pf  c\Pfc=<Z 


(29) 


Correct  Classifications  Vs  Incorrect  Classifications  in  “All  Created  Categories ” 

A  measure  of  the  performance  of  the  FHANN  in  all  categories  is  a  measure  of  the 
fraction  of  how  many  of  the  data  that  are  in  correct  or  incorrect  categories,  with  respect  to 
all  the  data  input  to  the  system.  These  are  defined  by  Eqs.  30  and  3 1 . 


2 ]D(CQ 

P"  ~  y  D(TC) 

Z»(/C) 

P'c  ~  Y,ditc)  (3I  ) 

Note:  The  following  inequality  may  exist  due  to  the  fact  that  several  potential  categories 
may  have  been  eliminated  prior  to  classification: 

D(TC)  >  D(CC)  +  D(IC) 


Total  Count  of  Incorrect  Categories 

This  measure  is  the  absolute  number  of  incorrect  categories  created.  It  gives  an 
indication  of  the  degree  of  useless  categories  which  the  network  creates  for  a  given  set  of 
conditions.  It  is  given  by  Eq.  32. 

p,c  =£/C  (32) 
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Correct  Vs.  Incorrect  Categories 

A  measure  of  the  performance  of  the  FHANN,  with  respect  to  the  number  of  correct 
categories  it  creates  and  the  number  of  categories  created,  is  measured  as  a  fraction  of  the 
number  of  correct  categories,  if  the  value  of  Nclass  is  known.  It  is  given  by  Eq.  33. 


Ycc 


Likewise,  for  number  of  incorrect  categories,  Eq.  34. 


(33) 


(34) 


3.2  Function  Descriptions 

The  analysis  was  performed  using  a  system  running  under  MATLAB  with  the  following 
functions: 

(1)  Anode:  Creates  an  output  node  for  ANN.  An  output  node  contains:  category  number 
(Cat),  count  of  adjustments  (Nj)  and  B.  Nj  is  initialized  to  one,  the  max  points  W  to  zeros 
and  min  points  V  to  ones.  Category  number  is  equal  to  the  number  of  categories. 

(2)  Volume:  Calculates  the  volume  of  a  hypercube.  The  volume  is  defined  as  the  sum  of 
all  sides  of  a  hypercube  (Eq.  21). 

(3)  Min-max:  Finds  minimum  and  maximum  values  of  each  input  feature  from  an  input 
data  file  (Eq.  9) 

(4)  Scale:  Scales  input  vector  to  be  within  [0,1]  (Eq.  9). 

(5)  NC:  Generates  node  constants  for  different  Nj’s  (Eq.  17). 

(6)  DOI:  Calculates  degree  of  inclusion  (Eq.  13). 

(7)  DPM:  Calculates  degree  of  perfect  match  (Eq.  15). 

(8)  Adjust:  Adjusts  vector  B  so  that  the  overall  hypercube  volume  is  within  limit 
(HVjmax).  It  requires  two  inputs:  Vector  B,  and  the  difference  between  the  max  and  the 
actual  volume  Delta  (Eq.  23). 

(9)  Learn:  Updates  and  returns  the  values  of  B  (which  contain  the  min  (V)  and  max  (W) 
of  the  hypercube)  using  input  vector  A. 

(10)  Trans:  Transforms  input  data  into  a  membership  function  R  using  DOI  and  DPM 
(Eq.  16a). 

(11)  Test:  Performs  a  match  test  (Eq.  18)  on  the  input  vector  and  category  input 
hypothesis. 

(12)  Stat:  Calculates  the  statistics  of  the  output  and  does  some  output  routines. 
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3.3  Main  Program  Description 


Interactive  input  values  are  requested  for  Vigilance,  Gamma,  Learning  Adjustment  Factor 
r,  K  (in  Eq.  16b)  and  maximum  hypercube  volume  0max. 

The  input  data  file  which  contains  the  feature  vectors  is  then  scanned  using  Function 
Min_Max  to  obtain  FmslfFmin  the  maximum  and  minimum  values.  These  are  used  to  scale 
the  input  data  set  using  the  Function  scale. 

The  first  data  set  is  read  and  classified  as  category  one.  An  initial  output  node  is  created 
for  this  data  set  using  Function  Anode.  Once  a  node  is  created,  the  number  of  nodes 
(Num.)  is  incremented  by  one.  A  scan  loop  reads  input  feature  sets  from  the  input  file 
until  an  end  of  file  is  detected. 


Inside  the  scan  loop,  data  sets  are  read  and  scaled.  The  scaled  input  data  goes  through  the 
transform  layer  using  Function  Trans  which  takes  the  input  vector  and  generates  a 
membership  vector  R.  Values  of  vector  R  are  then  sorted  in  ascending  order.  The 
category  node  which  has  largest  R  value  is  the  current  winner. 


Figure  5. 

FHANN  Functional  Block  Diagram 


The  input  data  set  is  set  at  the 
winning  category  node.  A  match 
function  is  performed  against  the 
input  data  and  the  winning 
category’s  maximum  and  minimum 
points  using  the  Function  test. 

If  it  matches,  then  a  hypercube 
volume  test  is  performed  to  see  if  it 
is  within  the  specified  maximum 
hypercube  volume.  If  not,  the  min 
and  max  points  are  adjusted.  The 
winning  category  node  learns  the 
input  data  through  the  Function 
learn.  If  it  does  not  match,  that 
winning  category  node  is  disabled 
and  the  category  node  with  second 
largest  R  value  is  activated. 

The  process  continues  until  the  last 
category  node  is  encountered  with 
no  match.  At  this  point,  a  new 
category  node  is  created  for  that 
input  data  set. 
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Figure  5  shows  the  Main  program’s  Functional  block  diagram.  For  a  more  detailed 
description,  see  the  Main  program’s  flow  chart  in  Figure  6. 


Learn  ( a , B , r , hd-max ) 


Create 

category 


Adjust (B , delta) 


Y :  Exit 
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3.4  Testing  Methodology 


The  FHANN  was  tested  using  a  parametric  testing  method.  A  set  of  baseline  values  was 
chosen  and  the  program  was  run  while  varying  a  single  parameter  in  the  baseline  and 
keeping  the  others  constant.  The  purpose  was  to  see  the  effects  of  one  variable  on  the 
output  of  the  network.  The  data  files  that  were  used  as  inputs  were:  iris.dat  (a  3-feature, 
150-cases  of  the  Iris  flower  classification  set)  and  WDBC.dat  (a  2-feature,  569-cases  of 
the  Wisconsin  Diagnostic  Breast  Cancer). 

For  each  run,  the  following  measures  of  performance  were  generated  and  graphed: 

-  Percentage  of  correct  classification  in  all  classes  (Eq.  28) 

-  Percentage  of  correct  classification  in  correct  classes  (Eq.  30) 

-  Number  of  incorrect  classes  formed. 

From  results  of  the  parametric  test,  the  best  value  for  each  parameter  was  chosen  to 
maximize  the  output  of  the  network. 

For  Iris.dat,  the  following  optimal  output  was  obtained: 

-  Percentage  of  correct  classification  in  all  classes:  76% 

-  Percentage  of  correct  classification  in  correct  classes:  79% 

-  Number  of  incorrect  classes  formed:  1 

when  Gamma  =  5;  Vigilance  =  0.7;  r  =  0.18;  k  =  0.1;  Hvmax  =  1.5 

For  WDBC.dat, 

-  Percentage  of  correct  classification  in  all  classes:  88% 

-  Percentage  of  correct  classification  in  correct  classes:  90% 

-  Number  of  incorrect  classes  formed:  1 

when  Gamma  =  5;  Vigilance  =  0.6;  r  =  0. 1 ;  k  =  0. 1 ;  Hv  max  >  0.5 

Figures  7  and  8  show  a  sample  run  of  the  output  from  the  network.  A  confusion  matrix  is 
generated  for  each  case  where  columns  are  the  actual  input  categories  and  rows  are  the 
output  categories  formed.  Fig.  9  plots  the  percent  of  correct,  number  of  correct 
classifications,  and  number  of  correct  categories  versus  vigilance  for  both  iris.dat  and 
WDBC.dat. 

3.5  Comparison  Between  SFAM  and  FHANN 

Simplified  Fuzzy  ART  Map  (SFAM)  [17]  has  higher  percentage  of  correct  classifications 
than  FHANN  in  the  Iris  case  and  about  the  same  in  WDBC  case.  However,  the  numbers 
of  incorrect  classes,  N_incor,  are  much  higher  for  SFAM  over  FHANN  in  both  cases. 
This  occurs  because  SFAM  is  more  scattered  than  FHANN  as  it  is  more  sensitive  to 
changes  in  the  input  features.  FHANN  appears  more  stable,  displaying  greater 
insensitivity  to  noise  in  the  input  features 
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INPUT  DATA  FILE:  iris.dat 

Vigilance  =  0.700000 
Gamma  =  5 .000000  ;  k  =  0. 1 00000 
Learning  adjustment  factor  r=  0.180000 
Maximum  hypercube  volume  =  1.500000 

Confusion  matrix: 

Out  1  2  3  Total  Percent 


1  50  0  0  50  0.33 

2  0  29  43  72  0.48 

3  0  21  2  23  0.15 

4  0  0  5  5  0.03 


Total:  50  50  50  150 

Percent:  1.00  0.42  0.86  1.00 

Number  of  correct  classifications:  1 14 
Number  of  incorrect  classifications:  36 
Percentage  of  correct  in  all  classes:  76/100 
Percentage  of  correct  in  correct  classes:  79/100 
Percentage  of  correct  classes:  100/100 
Number  of  incorrect  classes:  1 

Figure  7.  FHANN’s  Optimal  Output  for  Iris.dat 


INPUT  DATA  FILE:  WDBC.dat 

Vigilance  =  0.600000 
Gamma  -  5.000000  ;  k  =  0.100000 
Learning  adjustment  factor  r  -  0.100000 
Maximum  hypercube  volume  =  1.000000 

Confusion  matrix: 

Out  1  2  Total  Percent 


1  157  16  173  0.30 

2  39  341  380  0.67 

3  16  0  16  0.03 


Total:  212  357  569 

Percent:  0.74  0.96  1.00 

Number  of  correct  classifications:  498 
Number  of  incorrect  classifications:  71 
Percentage  of  correct  in  all  classes:  88/100 
Percentage  of  correct  in  correct  classes:  90/100 
Percentage  of  correct  classes:  100/100 
Number  of  incorrect  classes:  1 

Figure  8.  FHANN’s  Optimal  Output  for  WDBC.dat 
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#  Of  Incorrect  Categories  Vs.  Vig.  -  WDBC 


Percent  of  Correct  Vs.  Vigilance  -  WDBC 


Q1  02  0.3  Q4  05  0.6  Q7  0.8  0.9 

Vfglanoe 


Figure  9.  FHANN  and  SFAM  comparison  (Contd) 


4  SPEAKER  RECOGNITION  TEST 

This  section  describes  specific  testing  of  the  FHANN  for  recognition  of  speakers. 

4.1  Introduction 

Speaker  recognition  from  speakers’  voices  requires  a  clustering/classification  process 
which  can  group  signal  feature  representations  of  the  voices  into  reliable  speaker  groups. 
The  clustering/classification  should  be  able  to  form  any  number  of  classes  dynamically, 
and  tolerate  the  noisy  and  overlapping  domain  of  speakers’  feature  vectors.  ART 
architectures  did  not  perform  well  during  prior  speaker  recognition  testing  [13,14,16]; 
hence,  the  fuzzy  hypercube  classifier  was  employed  in  this  test  in  an  attempt  to  overcome 
this  and  other  problems. 

There  were  two  speaker  data  sets  used  for  the  formal  testing  of  the  system,  the 
Switchboard  [22]  and  the  Greenflag  [23].  Switchboard  data  consists  of  26  speakers  and 
Greenflag  41  speakers  in  a  tactical  environment.  Both  data  sets  were  interfaced  using  the 
NIST/SPHERE  Y2.2  software.  Speaker  test  data  were  grouped  into  non-overlapping  sets 
of  8,  12,  and  16  speakers. 
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The  Feature  Processor  is  described  in  detail  in  [15].  Features  were  analyzed  for  two 
characteristics,  separability,  and  maximum  /  minimum  values.  For  an  introductory 
background  in  this  area,  see  Pellisier  [20],  This  report  deals  only  with  the  classification 
performance  of  sets  of  input  features  provided  by  the  Feature  Processor. 

There  were  a  number  of  fixed  and  varied  parameters  corresponding  to  specific 
subsystems  [15]. 

FHANN-Variable  Parameters 
Vigilance 

Maximum  Hypervolume 

Overall  System  Variable  Data  and  Parameters 
Test  Data  Sets 
Number  of  Speakers 

Number  of  correct  and  incorrect  classifications  per  Test  Set 

4.2  Speaker  Testing 

The  following  are  defined  parameters  which  were  measured  during  the  speaker  testing. 

1 .  Average,  Minimum,  and  Maximum  of  Total  Number  of  Actual  Speakers  Correctly 
Identified.  Using  P£c  from  Eq.  28,  the  average  is  defined  as: 

AT  v 


where  D(CC)  >  1  in  Eq.  28,  and  Nlesls  are  the  total  number  of  speaker  recognition  tests 
performed.  The  Minimum  and  Maximum  are  defined  as: 

Min(  Pec),  Max{PcC)over  all  Nlesls  (35) 

Figures  10  and  1 1  both  display  the  Average,  Minimum,  and  Maximum  of  P^  as  a 
function  of  the  vigilance  parameter  and  the  maximum  hypervolume  [within  a  small  range 
of  values.] 
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Figure  10.  Percent  of  Correct  Classification  vs. 
Vigilance  Values 


Percent  Correct 


Classification 


^HIMinimum 

l  I  Maximum 


-Mean 


Figure  11.  Percent  of  Correct  Classification  vs. 
Hypervolume  Values 

Percent  Correct 
Classification 


I  Minimum 
]  Maximum 
-Mean 


2.  Average,  Minimum,  and  Maximum  of  Total  Number  of  Actual  Speakers  Incorrectly 
Identified.  Using  Pfc  from  Eq.  29,  the  average  is  defined  as: 


is 


C 

1C 


pc  = 

ric 


AT 


(36) 


where  D(IC)  >2  in  Eq.  36.  The  Minimum  and  Maximum  are  defined  as: 


Min(Pjc),  Max  (P,c)  over  allN,esls  (37) 

3.  Average,  Minimum,  and  Maximum  of  Spurious  Categories1  Generated.  Test  values 

are  generated  using  P,NC  Eq.  33  with  the  condition  D(IC)  <  2 : 


I  pi 


(38) 


1  Spurious  Categories  are  speaker  categories  with  a  maximum  of  1  or  2  entries  over  all 
time.  This  is  extended  to  time -dependent  spurious  categories  where  the  entry  count 
decreases  fractionally  and  is  proportional  to  age. 
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where  D(IC)  <  2  in  Eq.  38  and  39.  The  Minimum  and  Maximum  are  defined  as: 

Min(Pjc),  Max(Pjc)over  allNlesls  (39) 

Figures  12  and  13  display  spurious  speaker  category  creation  in  the  network  as  a  function 
of  vigilance  parameter  and  maximum  hypervolume. 


Figure  12.  Number  of  Spurious  Categories  vs. 

Number  of  Spurious 

categories  Vigilance  Threshold 


Number  of 
Spurious 

Categories  Figure  13.  Number  of  Spurious  Categories  vs. 

Maximum  Hypervolume 

15 
13 
11 
9 
7 
5 
3 
1 
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Maximum 

Hypervolume 
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Mean 


4.  Average,  Minimum,  and  Maximum  of  the  Number  of  Correct  Speaker  Categories  is 
defined  as: 


N„ 


Min(PcNc), 


Max(PcNc)over  all Ntests 


(40) 


Figures  14  and  15  show  the  development  of  correct  speaker  categories  for  the  8  speaker 
test. 
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Number  of  Correct 
Speakers  out  of  8 
Maximum 


Figure  14.  Number  of  Correct  Categories  vs. 
Vigilance  Threshold 


Number  of  Correct 
Categories  out  of  8 
Maximum 


Figure  15.  Number  of  Correct  Categories  vs. 
Maximum  Hypervolume 


Summarized  test  results  for  the  FHANN  performance  are  shown  in  Tables  1  and  3  for  the 
8  speaker  tests,  and  Table  2  for  the  12  speaker  tests. 


TABLE  1.  FHANN,  8-Speaker  Test 


Test  Data  Set  for  8 

Average  Number  of 

Average  Number  of 

Average  Number  of 

Speakers 

Correct  Categories 

False  Categories 

False  Categories 

Generated  (8  max) 

Generated  ( 8  max) 

Deleted  per  Data 

Set 

Switchboard  May  95 

6.29 

0.29 

1.86 

Greenflag 

6.57 

0.23 

5.77 

TABLE  2.  FHANN,  12-Speaker  Test 


Test  Data  Set  for 

Total  Number  of 

Total  voiced  Speaking 

Overall  Correct 

12  Speakers 

Speakers  in  Test 

Time  (mins) 

Classification  (%) 

Switchboard  May  95 

12 

67.25 

Greenflag 

23 

68.75 
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TABLE  3.  FHANN,  Summary  of  Test  Results 


Test  Data  Set  for  8 

Total  Number  of 

Total  Voiced 

Overall  Correct 

Speakers 

Speakers  in  Test 

Speaking  Time 
(hrs) 

Classification  (%) 

Switchboard  May  95 

26 

2.69 

69.7 

Greenflag 

41 

2.96 

70.3 

Percent  Correct 
Classification 


Figure  16.  Percent  correct  vs.  Greenflag  Test  Groups 


Percent  Correct 
Classifiction 


Test  Groups 


Figure  17.  Percent  correct  vs.  Switchboard  95  Test  Groups 


A  breakout  of  the  Switchboard  and  the  Greenflag  individual  test  groups  is  shown  in 
Figures  16  and  17.  Here  the  results  are  given  for  each  group  individually,  with  associated 
absolute  minimum,  maximum  and  mean.  Values  are  derived  using  Eq.  34  and  Eq.  35. 

The  overall  system  test  results  are  shown  in  Table  4.  This  includes  all  speaker  groups. 
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TABLE  4.  Overall  System  Test  Results 


Test  Data  Set  for 
all  Speaker 

Groups 

Total 

Number  of 
Speakers 

Total  Voiced 
Speaking  Time 
(hrs) 

Overall 

Correct 

Classification 

(%) 

Standard 

Deviation 

(avg) 

Maximum- 

Minimum 

(avg) 

Switchboard  [14] 

26 

3.0 

66.9 

5.0 

14.5 

Greenflag  [15] 

41 

66.6 

6.6 

13.4 

4.3  Test  Results  Discussion 

The  speaker  tests  performed  in  the  previous  section  will  be  described.  They  will  be 
grouped  into  ANN  Tests  and  Overall  System  Test.  The  Overall  system  test  included  the 
testing  of  the  Feature  Processor  with  the  FHANN. 

4.3.1  FHANN  Specific  Tests 

FHANN  parameters  modified  during  testing  were  maximum  hypervolume  and  the 
vigilance  parameter.  In  Figures  10  and  11,  the  effects  of  the  hypervolume  (HV)  and 
vigilance  on  the  system  performance  are  demonstrated  for  a  very  limited  set  of  values. 
The  bar  charts  indicate  that  the  HV  mean  value  does  not  change  much,  but  the  spread 
between  the  maximum  and  minimum  increases  with  increasing  hypervolume.  Thus,  one 
would  like  to  keep  the  HV  as  small  as  possible,  while  maintaining  an  acceptable 
classification  performance  level. 

Spurious  categories  are  displayed  in  Figures  12  and  13.  Here  an  increase  in  vigilance 
produced  an  increasing  number  of  spurious  categories,  as  well,  and  wider  swings  on  the 
maximum  and  minimum  values.  This  makes  sense  since  increasing  the  vigilance 
produces  a  more  “specific”  network,  and  hence  more  extra  category  nodes  would  seem  to 
be  created  under  these  conditions.  The  effects  of  hypervolume,  in  contrast,  seemed  to 
have  little  effect  on  the  spurious  category  creation. 

The  bar  chart  in  Figure  14  shows  a  slight  decrease  in  the  number  of  correct  speaker 
categories  with  an  increase  in  vigilance,  but  a  narrowing  of  the  max/min  values.  For  the 
hypervolume,  in  Figure  15,  there  is  a  slight  decrease  also,  but  not  as  clear  as  that  for 
vigilance. 

Figures  10  to  13  relate  vigilance  and  hypervolume  to  the  number  of  categories  generated 
in  the  category  layer  in  the  network,  both  correct  and  spurious.  In  the  case  of  vigilance, 
there  is  a  slight  downward  trend  to  performance  for  a  correct  number  of  categories,  and  a 
strong  positive  increase  in  the  number  of  spurious  categories  generated  as  it  increases. 
In  the  case  of  hypervolume  maximum,  increasing  values  have  little  effect  on  the  number 
of  correct  categories  and  spurious  categories. 
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Note  that  the  tests  should  not  be  used  to  draw  generalizations  over  the  entire  data  space 
due  to  the  fact  that  data  sets  are  narrow  in  range. 

4.3.2  Overall  System  Tests 

Overall  testing  results  are  shown  in  Figures  16  and  17  and  Tables  2  to  4.  In  Figures  16 
and  17,  system  performance  is  plotted  for  each  speaker  group,  and  is  shown  for  its  mean, 
minimum,  and  maximum.  The  results  are  synopsized  in  Tables  2  and  3,  giving  the 
standard  deviation  averaged  over  all  groups  for  each  group,  as  well  as  the  maximum  to 
minimum  value  spread  averaged  over  all  the  groups. 

From  these  data,  it  can  be  seen  that  Greenflag  had  a  smaller  minimum  to  maximum 
spread,  and,  with  the  exception  of  group  number  7,  all  appear  well  behaved.  In  the 
switchboard  case,  the  spread  is  much  more  in  all  groups  with  group  number  13  the 
greatest.  However,  the  switchboard  data  was  still  more  well  behaved  and  better  clustered 
as  is  shown  by  its  smaller  standard  deviation  in  Table  4. 

The  performance  of  the  test  groups  is  nearly  identical  at  67%  for  an  8  speaker  group 
maximum. 


5.  CONCLUSION 

A  seven  layer  neural  network  architecture  is  described  which  performs  a  hypothesize  and 
matching  test  between  an  input  vector  and  a  fuzzy  hypercube  category  representation  of 
the  input  vectors. 

The  network  has  a  variable  hypervolume  limit  to  accommodate  noisy  feature  hypercubes. 

Testing  was  performed  to  compare  the  fuzzy  hypercube  classifier  with  fuzzy  ART  using 
Iris  Flower  and  breast  cancer  standard  data  sets.  The  fuzzy  hypercube  classifier  displayed 
better  tolerance  to  noise  in  these  tests. 

The  fuzzy  hypercube  classifier  was  also  tested  with  Switchboard  and  Greenflag  data  sets. 
The  performance  for  8  speaker  groups  is  67%  overall  correct  classification. 
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Appendix  A1  -  MATLAB  Toplevel  Code  Listings  of  SFAM 


%*************************************************** 
%*  Simplified  Fuzzy  ArtMap  (SFAM)  program  * 

%*  Programmer:  Hai  Phu  * 

%*  Date:  22/08/96  * 

%*  * 
%*  Input  data  file:  Iris.dat  * 

%*  Output  data  file:  Out.dat  * 

0^**************************************************+ 

clear; 

Infile  =  ’h:\matlab\iriscnvt.dat’; 

Outfile  =  'hitoatlabtefamVoutdat'; 

fld_in  =  fopen(Infile);  %  Open  a  input  data  file 

fidout  =  fopen(Outfile,  V);  %  Open  an  output  file 

today  =  date;  time  -  clock; 

fprintf(fid_out,'DATE:  %s  %2d:%2d\n', today ,time(4:5)); 
fprintf(fid_out, ’INPUT  DATA  FILE:  %s  \n\n',  Infile); 

%  ********************  Start  of  program  ******************** 
m  =  iris_mm(fid_in); 

1  =  0; 

Vigilance  =  .1 
while  Vigilance  <=  .9 
1  =  1+1; 

status  =  fseek(fid_in,0,'eof  );  %  set  indicator  to  end  of  file 

eof  =  ftell(fid_in); 

status  =  fseek(fidjn,0,’bof );  %  reset  indicator  to  beginning  of  file 

%  Initialization 

Num_nodes  =  0;  pos  =  0; 
wrong  =  0; 
count  =  1 ; 

Category  =  0; 

******************  data  set  ******************* 

Data  =  fscanf(fid_in,’%3f  %3f  %3f  %3f  %d  /n'); 
a  =  Data(l:4)'; 

In_Cat(count)  =  Data(5)  +  1; 

Out_Cat(count)  =  1 ;  %first  data  is  categorized  as  one 
pos  =  ftell(fid_in); 

b  =  scale(a,m);  %  scaling  the  input 

Num  nodes  =  Num_nodes  +  1 ;  %  create  an  initial  output  node. 

[Category(Num_nodes) ,  w(Num_nodes, :)]  =  fhode(b,Numjiodes); 
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o/o**********  Reading  input  data  till  end  of  file  ************* 
while  pos  <  eof 
count  =  count  +  1 ; 

Data  -  fscanf(fid_in,'%3f  %3f  %3f  %3f  %d  /n'); 
a  =  Data(l:4)'; 

In_Cat(count)  =  Data(5)  +  1 ; 
pos  =  ftell(fid_in); 

b  =  scale(a,m);  %  scaling  the  input 

%  ***************  Hypothesize  input  data  ****************** 
[node,i]  =  hypo(Num_nodes,w,b); 

%  ******************  Jest  hypothesis  ********************** 

[w,switch,Out_Cat(count)]  =  Test_hyp(node,i,w, Vigilance); 

%* ****************  Create  new  category  ******************** 

if -switch  %  if  never  seen  this  category  before 

Num_nodes  =  Numnodes  +  1 ;  %  then  create  an  output  node. 
[Category(Num_nodes)  ,  w(Num_nodes, :)]  =  fnode(b,Num_nodes); 
Out_Cat(count)  =  Num_nodes; 
end; 

end  %  end  while  eof 

o/0********************  Output  routine  ********************** 

fprintf(fid_out,'\nVigilance  =  %f\n', Vigilance); 

[Pc_ac(I)  Pc_cc(I)  TJncor(I)]  =  stat2(In_Cat,Out_Cat,fid_out); 

%print  out  statistics 
Values(I)  -  Vigilance; 

Vigilance  =  Vigilance  +  .1 

clear  w  Category  node;  %  clear  out  all  vars  for  next  run 


end 


%  end  while  Vigilance 


plots(  V  alues  ,Pc_cc,T_Incor,Pc_ac) 

fclose(fid_in); 

fclose(fidout); 
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Appendix  A2  -  MATLAB  Toplevel  Code  Listings  of  FANN 


o^*  ********************************************** 

%*  Fuzzy  Hypercube  Artificial  Neural  Network  program  * 

%*  Programmer:  Hai  Phu  * 

%*  Date:  22/10/96  * 

%*  * 

%*  Input  data  file:  Iris.dat  * 

%*  Output  data  file:  Out.dat  * 

%*  ********************************************** 

clear; 

Infile  =  'h:\rn  atlab\iriscnvt.dat'; 

Outfile  =  'h:\matlab\h_annl\out.dat'; 

fid  in  =  fopen(Infile);  %  Open  a  data  file 

fidout  =  fopen(Outfile,V);  %  Open  an  output  file 


today  =  date;  time  =  clock; 

fprmtf(fId_out,T)ATE:  %s  %2d:%2d\n', today  ,time(4:5)); 
fprintf(fid_out, 'INPUT  DATA  FILE:  %s  \n\n'}  Infile); 


o/Q  ********************  Qf  program  ******************** 

0^* *********************  User  inputs  ************************ 
runs  =  0; 


Vig  =  input('Enter  a  value  for  vigilance  > '); 

Gamma  =  input('Enter  a  value  for  gamma  >  '); 

r  =  input('Enter  the  Learning  Adjustment  Factor,  r  > '); 
k  =  input('Enter  a  value  for  k  >  '); 

Hv_max  =  input('Enter  the  maximum  hypercube  volumn  > '); 


while  Hv  max  <=1.5 


runs  =  runs  +  1 

<^* ********************************************************* * 
m  =  mm_iris(fid_in); 


status  =  fseek(fid_in,0,'eof );  %  set  indicator  to  end  of  file 

eof  =  ftell(fid_in); 
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status  =  fseek(fid_in,0,'bof);  %  reset  indicator  to  beginning  of  file 


%  Initialization 


Num  =  0; 
count  =  1 ; 


o/0*******************  Read  gj.st  jata  set  ******************* 

%read_iris; 

Data  =  fscanf(fid_in;%3f  %3f  %3f  %3f  %d  /n'); 
a  =  Data(l:4)’; 

In_Cat  =  Data(5)  +  1; 
pos  =  ftell(fidjin); 

a  =  scale(a,m);  %  scaling  the  input 

Num  =  Num  +  1 ;  %  create  an  initial  output  node. 

[B(Num,:),  Nj(Num)]  =  anode(a); 

Out_Cat(count)  =  1 ;  %first  data  is  categorized  as  one 

0/o**********  Reading  input  data  till  end  of  file  *********** 

while  pos  <  eof 
count  =  count  +  1 ; 

Data  =  fscanf(fid_in/%3f  %3f  %3f  %3f  %d  /n'); 
a  =  Data(l:4)'; 

In_Cat(count)  =  Data(5)  +  1; 
pos  =  ftell(fid_in); 

a  =  scale(a,m);  %  scaling  the  input 


%  ***************  Transform/Fusion  layer  ****************** 
for  category  =  1  :Num 

R(category)  =  Trans(B(category,:),a,Nj(category), Gamma, k); 
end; 

%  **************  Hypothesize  and  test  layer  *************** 

%  Test  an  hypothesis;  function  testjiyp 
[winner, node]  =  max(R); 

Adj  =  Nc(Nj(node))*  Vig; 

Out_Cat(count)  =  node; 

Match  =  winner  >  Adj; 

0/q  **********************  Learn  ************************ 


if  Match 
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B(node,:)  =  leam2(a,B(node,:),r);  %  Hypercube  learning 
Nj(node)  =  Nj(node)  +  1 ; 

o/o  **********************  Adjust  *********************** 
Shv  =  volume(B(node,:)); 

if  Shv  >  Hvmax  %  if  volume  exceeds  Hvmax  then 

delta  =  (Shv  -  Hv_max)/Num 

B(node,:)  =  Adjust(B(node,:), delta);  %  adjust  Hypervolume 
end; 

0/q* ****************  (Create  new  category  **************** 

else  %  if  never  seen  this  category  before 

Num  =  Num  +  1 ;  %  create  an  output  node. 

[B(Num, :),  Nj(Num)]  =  Anode(a); 

Out_Cat(count)  =  Num; 

end; 

end  %  end  while  eof 


q/q* *******************  Qutput  routine  ********************** 

fprintf(fid_out,'\nVigilance  -  %f\n',Vig); 
fyrintf(fid_out, 'Gamma  =  %f ;  k  =  %f\n’, Gamma, k); 

^)rintf(fid_out, 'Learning  adjustment  factor  r  =  %f\n',r); 
fj)rintf(fid_out, 'Maximum  hypercube  volume  =  %f\n',Hv_max); 

%save  parameters  for  plotting  purposes 

[P_ac(runs),P_cc(runs),N_icc(runs)]  =  stat(In_Cat,Out_Cat,fid_out); 

Values(runs)  =  Hv_max; 

Hv_max  =  Hv_max  +  0.25; 

clear  B  category  Nj  R;  %  clear  out  all  vars  for  next  run 

end  %while 

plots(Values,P_cc,N_icc,P_ac,'Hv_max’); 

saveit  =  input('Do  you  want  to  save  all  working  variables  ?’,'s'); 
if  saveit  =  'Y*  |  saveit  =  y 

filename  =  input('Enter  filename  ->  '/s'); 
filename  -  ['h:\matlab\sfam4V  filename]; 
save(filename); 
end; 

fclose(fid_in); 

fclose(fidout); 
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